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ABSTRACT . ; xe ‘ 
Designed for use with the Research for Better Schools 
Aareer education program, this evaluation planning sanual focuses on. 
procedures and issues central to planning the evaluation of an 
educational. program. Following a statement on the need for , 
evaluation, nine sequential steps«for evaluation planning are ; 
discussed. The first two steps, prodram definition and evaluation’ “ 
questions, serve as a guide for-developing the intended scope of the 
program and evaluation. The next five steps review evaluation 
pethodology in terms of (1) the statement of hypotheses, (2) 7 
selection of subject groups, (3) selection of instruments, (4) - 
creation of a data system, and’ (5) design of an analysis plan-~The 
final two steps focus on planning concerns in evaluation Hepertiag 

and cost projections (LRA) ‘ 
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‘INTRODUCTION, st | Gees Se 


Evaluation has been a eaistlniing component in ile development of vas 
Career Education. The -evaluation findings have provided a tseful source of 
information in refining the program as well’ as offering: evidence of program 
effectiveness to participants, sponsors, potential adopters and other members of the 
educational community. 

_ Since RBS Career Education has beconite available for adoption by spublic 
school districfS, a series af materials and services has been prepared to assist adopters 
in alga implementing and evaluating the program. This series includes: 


. 
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Materials * Services 
-Evaluation Planning Manual — Evaluation Technical Assistance 
, Instrument Service Guide| = +— Instrument and Scoring Service 
Analysis Service Guide. -— Analysis Service \ 
Program Monitoring Manual © — . Evaluation Technicaf Assistance : 


The materials are intended to assist in evaluation planning and design, while the 

services make. available Research for Better Schools’ evaluation: systems and 

.&cpertise to support iniplementation. These materials and services are described 
re completely in the RBS Evaluation Package Overview and can be obtained from 
Research for Better Schools. . 

This RBS Evaluation Planning Manuat, one element in the series, aioe on 
procedures and issues central to planning the evaluation of an educational | program. 
Evaluation planning is discussed in a framework of major sequential steps. After a’ 
statement on the need for evgluation, the inténded scope is developed by addressing 


‘planning activities in program ‘definition and evaluation questions. Evaluation 


methodology then is reviewed in enue of hypotheses, subject groups, ‘ingtruments, 
data systems and‘analyses. ‘Concluding se@tions treat: Pang cagcerns in evaluation 

The manual has been designed to provide guidance ji in planning educational 
program evaluation. It has been developed jn the career education context, but the 
evaluation concerns addressed are common ‘to other programs as well. Assumptions 


about loeal conditions have been avoided in the interest of providing broad coverage 


a 


of generic topics. Applying planning suggestions to any specific sregrami requires 
consideration of these local conditions. Research for Better Schools can provide 
evaluation technical assistance to aid in the formulation of individual designs. 


a. @ 


- NEED FOR EVALUATION. 


As it has been developed for RBS Career Education, evaluation functions to ‘ 


meet, several major information needs, which are categorized as student diagnosis, 
program planning and demonstration of program éffects. The content of each of 


these .categories and the relevance of the evaluation ‘materials to them will be. 


described briefly. a 
Since RBS Career Education emphasizes the individual treatment of students, 
it is important to have detailed and accurate information about each student in the 


program. Such information can be+an aid’ in placing students, planning their . 


experiences.and providing personal guidance. A major criterion for selecting the 
instruments recommended in the evaluation package (see RBS Instrument Service 


Guide) was their ability to yield ustful; individual data. For instance, the 


Self-Directed Search includes occupations considered, self-estimates of interests and 
_ competencies and prescriptive summaries. The Student Attitude Survey capsulizes 


student attitude toward school, work, self and others. The Comprehensive Tests of 


Basic Skills reflect functional levels in reading, mathematics, language and study 
skills. The Stident Demographic Data Questionnaire gives basic information on 
student background variables. These instruments or others which may be selected 
provide the information for assembling a student profile that becomes a part of each 
student’s record and may be tised to chart his or her course. It will indicate interests, 
strengths, weaknesses and perceptions which are helpful in designing experiences for 
individuals. } ’ 

The same data gathered for individual students may serve a program planning 
function when summarized for all students in the program. At this level, student 
career interests, for example, help to determine the range ‘of career. experiences 
which should be provided. Group needs in basic skills suggest the natde and extent 


of academic content and materials which would be appropriate. Affective needs also " 


¢ 


can be identified, and program tia can be focused to meet them: 
_ Surveys of participant opini 
They measure the perceptions of students, pdrents and community participants 


regarding the program and are intended to gather opinions about its various aspects . 


and the success of its implementation. Their design allows adaptation to each site’s 
needs and interests. Information gathered in this way is helpful in ass, program 
conduct from the viewpoint of the people involved, and the r&ults often have 
planning implications. 


likewise are relevant to program planning. ~ 
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; ; Finally, the effects ‘of the progtam on students may be sverige by 
administering’ instruments in a research’ paradigm and analyzing student develop- . 

“| ment. These procedures yield information about student progress on the selected 

. measures during their program experiences. Changes: observed among project 

_ participants then may be compared with changes over the same time period among - 
similar students who have not been engaged in the program. These comparative | 

__ analyses make it possible to draw inferences of relative progr impact on student 

,| | development. Student effects are tested by using statistical analyses in a hypothesis - 
bse ra framework r representing the desired effects of the program. 

‘* Similar kinds of student data thus can be used in several ways if proper 
evaluation planning is accomplished while program implementation is being 
designed. Needs in student diagnosis, program planning and monitoring. and ‘the 

-demonstration of student effects can be identified through an evaluation com- 
ponent. In this way evaluation activities can serve program opetations, development, 
administration and research. 

Once the importance of program evaluation. has been endorsed, the next step 
is to ) plan an evaluation component which will be effective in meeting the néeds. The - 
4i following sections of this manual address the major steps in the evaluation process 

from a design perspective. Issues central to planning the evaluation are’ discussed, 

Setar _ ad important decision points are indicated detailed implementation concerns and 

problems have not ‘been included because their dependency on individual conditions 
prevents comprehensive and concise treatment here. ©) md \ 
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PROGRAM DEFINITION a8 ae Re 


- The first step in evaluation planning is preparing an accurate description of the 
program which is to be evaluated. Three levels or types of description are necessary ” 
for evaluation purposes: a program overview, is iat of the program , and 
operational strategies. . 

/ a The program overview describes the components sallich have been planned for / 
plementation, how they are organized and who is to participate, .It serves as a ° 


broad definition of the pe of the program .and ‘the context, within which the ae 
evaluation is to be conducted, , 
Program objectives are statements of what the program intends to dccomplish. 


They may be a-combination of process objectives and product objectives. Process _ 
objectives relate to the wae Sa or adequacy of implementation, while product - 
objectives are concerned with the outcomes or effects of the’ program. An example 
of a process objective would be, ‘to provide three career exploration experiences for 
each enrolled student.” A prodiict objective might be, “to increase the career 
maturity of participating students,” ‘It is important to develop a list of all objectives.” |. . 
which the program is intended to meet and to define them as specifically as.possible. 5 
Sometimes it is necessary to,have some objectives that are more abstract than others,” 
: but it should be understood that they will be more difficult to evaluate. The 
‘statement of objectives, then, defines the intent of the program in pe terms and 
-establishes expectations of how the program will perform. |. as 
Once the objectives have been identified, they can be grouped and assigned Se mis 
priorities. Grouping should be’ done by objective type: Process or product, ‘cognitive : 
ee ae short-term or long-term, etc. The grouping can be done in any manner ° ro \ ; 
ich ‘results in an understandable framéwork of objectives that represent the ‘Nw 
interest areas of program associates and ‘sponsors. Relative priorities then should be y ‘ 
w assigned to guide the ‘allocation, of evaluation resources. Although the evaluation, 
must be addressed to all” objectives, some: among them may merit differential : 
‘attention depending on their relative importance to the program, the probability of 
obtaining conclusive results, the potential decision-making value of findings or other 
factors. Establishing priorities ‘among objectives is helpful in clarifying any existing 
hierarchies of objectives and suggesting relative emphases in the evaluation design. : : 
Operational strategies link objectives to those program elements which are 
designed to accomplish them. Each objective should be described in terms of tHe cae, 
operational procedures designed to attain it. Such descriptions serve to assure that 
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' stated objectives actually are associated with specifiable project activiti@. Objectives 


which cannot be tied to at least one activity ‘signal a problem which requires a 
redesign of either the program content or the objectives involved. 

The completion of these three descriptive tasks results in a definition of the — 
program in terms of jts scope, intent and programmatic process. This. combined 
definition becomes the basis of the evaluation effort in that it circumscribes that 
which is to be evaluated and also establishes expectations and accountabilities for 
the ‘program. Defining’ the program is a process that’ should include planners, 
implementers and evaluators. All project personnel should acknowledge and support 
the resultant statements of definition to ensure that everyone proceed{ on a’ 
common basis. 


. EVALUATION QUESTIONS 


if the evaluation effort is to meet ‘the needs of implementers, planners ‘and 
. sponsors, it must focus on ‘specific questions which are significant to the program. 

The process’ of defining the program yields « objectives, which are -necessary\ in 
formulating: evaluation questions, but the objectives themselves do not constitute 
such’ questions. Program ei ae are statements of intended educational out- 
comes; and the evaluation. process. requires translating these objectives into 
v hypothesized effects which can be empirically tested. The formulation of evaluation 
questions facilitates this translation process. ; 

Stating objectives is primarily the domain of program imiplenienters: and * 
planners: because they know the intentions of the program best. The statement of — 
hypotheses is the realm of evaluatots, who know the scientific -and technical issues 
best. Formulation of evaluation questions is the middle ground where all;program. 
associates, participate equally in exploring: i implications of objectives ‘and 
establishing the bases for developing hypotheses. This intermediate step is'a helpful 
process for assuring that the evaluation design fairly and completely represents 
program intentions, It also promotes’ interaction among: all staff i in laying the 
foundation for evaluation of the program. - ~ + ° 

Evaluation questions ‘are derived from the group of program objectives 
through exploration of their content and ‘implications. This ia process 
should include program implementers, planners, sponsors and evaluators, who 
should aim at specifying observable consequences associated with the objectives:and 
reasonable standards of success. Such specification permite: the formulation of 
evaluation questions. . pie, eign ® 

The development of evaluation questions may ‘be illustrated by using the 
sample objective, *“to increase the career maturity, of participating students.” - 
Examination of the intent behind this objective might yield ‘‘career planning skills” 
and “confidence in making a career choice” as the appropriate variables-represented 
by” the objective. Further, it might be.decided that the standard for judging success 
should be demonstrable Progress of students during their participation in the 
program. In this case -appropriate evaluation questions would be: ‘‘Do students - 
increase their confidence in making a career choice?” For both questions the 
demonstrated progress of students during their participation in the program would 


‘be the standard for jetiging whether the objective has been.met. 


‘ 
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Z GM ae a . “This procedure Meet be. followed ie ae program - abeoees Each. eee 

' eo ‘ will result in at least one evaluation | question, and many objectives, . upon 

fo exploration, will require more than one evaluation question to represent adequately = 
+ | --their intent. The process of. developing these évaluation questions ensures that the 

. aed “implications of program objectives have been examined, that the program intentions: 

| "| are reflected’ reasonably in the evaluation and that the groundwork for representa” a 


ur tive hybotheser® completed. =.” ak 
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STATEMENT OF HYPOTHESES " 


-. The collection: of évaluation questions displays the desired scope of the 
eeiluation component. The next get of tasks is concerned with establishing the. 
_means whereby these questions’can be answered reliably and validly. Although ways i 
will be found to address most questions adequately, it may be anticipated that some 
questions will have to be eliminated on technical or cost grounds. The Priorities 
established earlier will help in making these decisions. 
. Hypotheses are propositions or assumptions ‘constructed to draw out and test . 
the logical- or empirical consequences of the announced objectives as they represent 
the program “theory.” Formulating. hypotheses involves the refinement of evalua-’ 
tion questions into testable propositions, which necessitates establishing a standard 
of success for-each question. For example, an evaluation question might be: “Do 
. Students’ gain career planning skills through participation in the- program?” The most 
‘basic hypothesis in this .case would be: “Students will score higher at the end of the 
program year than at the beginning on the XYZ test of career planning skills.” The 
standard is-higher performance on a relevant measure over a year of exposure to the 
, program. The meaning of “higher” may be defined further as'some standard unit 
score gain or gains of any magnitude which are statistically reliable. : 
- This form of- hypothesis allows for determining changes which occur during 
- the intervals between gests,’ but it’ does not permit conclusions about whether the 
program was responsible for the changes. Other factors such as maturation, peer 
group. interactions,. media exposure and other events may have had some effect 
‘during the same time » period. The typical ‘method of taking these non-program 
.. influences irito ‘account is to compare the growth of program students with the 
_ progress of similar non-program student groups on‘the same measures. If this option 
is elected, hypotheses. then become comparative statements. The sample hypothesis 
. used above would become: “After exposure to the program; students in the program 
® «will score higher than comparisan studegts not ih the program on the XYZ test of | 
_areer planning skills.” The ‘term -“higher” again should be. defined in terms’ of 
-_ Statistical standards. - * . 
This process of refinement must be carried ¢ out for each evaluation question. It 
* will be found: that some questions will be mdr@amenable than others to restatement * 
as hypotheses. Comparative hypotheses, such as the career planriing example; lead to © 
ih most, definitive tests of. results and should be used wherever possible. Many 
ict a program ia such, as improved oe reading skills, vorational 
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ateinndss and others may be cast feinnsiedy ina comparative form. 

Some evaluation questions will not fit into a.comparative hypothesis paradigm 
because they relate only to program participants. Such questions are not appropriate 
for ‘non-program comparison groups. Examples would be: “Are student interests met 
by the program?” “Doés the business community. support the: program?” In such 
cases comparative hypotheses are not possible, and standards of success must. be 
established entirely within the program reference. Sample hypotheses might be: 


“Expressed student interests are matched by program activities at least 80 per cent 


of the time.” “Participating businesses and agencies recommend involvement to 
others in at least 80 per cent of the cases.” 

Testing both comparative hypotheses and within-program hypotheses requires 
__ acceptable subject groups, instrumentation ane statistical procedures as discussed 
‘below: - 


” ie 


SUBJECT GROUPS. 


. 


After hypotheses have been formulated, it is necessary | to select subject groups 
that can provide the data needed to test them..Evaluation is possible without . 


comparison groups, but tht usefulness of such results generally does not justify’ the 

expense of generating them. For the purpose of this manual, it will be assumed that 

comparative hypotheses are to be in¢luded. Two kinds of comparison group lege 
are discussed: true experimental and quasi-experimental.. “s, . ° 

_ The true experimental design requires that subjects be ‘randomly ssaigned to 

the experimental and comparison groups. The experimental subjects participate in 

the program, while the comparison group members are engaged in other activities 

which are distinct-from the experimental program. In most educational evaluations 

. the comparison groups are enrolled in a traditional curriculum or another cotnpesing 

rogram. " t) . 

* ‘Randomly assigning -subjects to. the experimental and comparison groups 

/ eliminates,the problem of selection bias, which typically. cenfounds other designs. 

Since each subject has. an equal chance of being assigned to either group, the 


likelihood of obtaining groups imbalanced on any characteristic is minimized. This - 


method presents the best ‘conditions for conclusively testing hypotheses because 
observed group differences in measured outcomes more likely will be due to 
program differences rather than possible differences in the groups themselves. 

: Random assignment usually is possible whére-the number of ‘Program 
applicants exceeds ‘the number that can -be adfhitted. In these. cases random 
assignment is: actually the fairest way of determining who should be enrolled i in the 
experimental program, Each applicant-has an equal chance, 

It° should be understéod that randomization precludes the passibility of 
selecting subject? on any special criteria unless such subjects are to be excluded from 
the program evaluation. A random assignment plan restricts the influence of staff on 
the composition of subject groups sq that energies are directed toward ensuring that 
the applicant pool fontains the desired target population mix. As desirable as 

‘random assignment/\s for evaluation purposes, it may be objectionable to those who 
seek to have certa& ipdividuals or groups in the experimental program- and could 

‘become an issue at the admjnistrative level. There is:no sure solution to conflicts of 

_ this’ ‘type; competing interests must be weighed. 

The. quasi-experimental design utilizes. comparison erouips which are not 


“random in their composition but which can be justified: as providing legitimate 
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comparative data. Such an 1 approach may be necessary either as a substitute for or . 


supplement, to a true experimental design. Substitution rhay be required where the 
appHcant pool is not large enough to form both’experimental and comparison. 
groups or where, administrative: considetations prectude randomization, Supple- 


mentation may be recommended where hypotheses call for comparisons with « 


identifiable: oups which cannot be constituted randomly from the applicant pool. 
-Examplés of jsuch. h groups. would be typical niga hee! students, worksdy students 
and school opouts. 

_ Whether Tac enseaneaea groups provide the only comparisons or ‘supple- 
mentary comparative data, the groups must be selected with great care to meet the 


needs of hypothesis testing. Criteria used in selecting the experimental groups must’ 


be. documented so that any’ resultant special characteristics can be- identified. 


- “Comparison groups not differing markedly from the experimental groups, but still _ 


, permitting ‘the desired comparisons, should be sought. Demographic, cognitive and 
affective‘ chatacteristics of all groups should be determined to the degree of 
conipleteness possible. The potential effects of. initial experimental-comparison 
group differences on the dutcomes to be measured should be estimated to ans a, 


‘ background for interpréting the final results. 


It must be recognized: that the quasi-expérimental design ows bi 
confidence in conclusions than the true experimental design since the: quasi-- 


experimental groups are more, likely to be different at the outset and these 
. differences may be suspected of affecting the evaluation results. Statistical 
_ procedures can compensate to % degree, but the design is inherently weaker. Serious 


consideration should be given to the advantages and disadvantages of the various 
subject group designs before a selection’ is made, and. _administrative, as well as 


_ evaluative, consequences should be ‘examined . carefully. Once groups have been 


constituted, changes will not be Beane within’ an eaperinental year, and they are 
often difficult between years. 

Students are. the ,principal sihietacle in most edteational programs, and ehis 
establishment of. student groups automatically creates parent groups. Other subject 
groups in the evaluation may consist of community resources, instructors, potential 


. adopters and others. The specific array of groups necessary is-defined by the range 


of hypotheses to be tested. Success in establishing ” apprepiisre aiect sors - 


determines the ability to test hypotheses. ; e ; 
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INSTRUMENTS 


Hypotheses determine what is to be evaluated; abject groups determine the 
samples with which hypotheses will be tested. The next step is to select instruments 
which reasonably can be expected to measure the hypothesized program effects. For 
each hypothesis at least one measure must be selected to represent the intended 
outcome. ‘Such measures may range from performance on a standardized test to 
opinions about aspects of the program. Indirect measures such as: attendance, 


assignment} completed and frequency of resource use’ also may be appropriate as. 


criteria for evaluating effects. 

A series of instruments, along with scoring sind interpretation packages, is 
available for use with RBS Career Education. These instruments are relevant to the 
measurement of career skills, life skills; basic skills and participant opinions. 
“ Depending on the scope of hypotheses; selection from among these instruments may 
suffice or additional measures may be needed. The instrumentation materials are 
described in detail in the RBS Instrument Service Guide. 


Just as the selection of instrument content must be keyed to project objectives 


and hypotheses,-the schedule of administration must be timed to permit the desired 
analyses. Some hypotheses may require data gathered at one time only, as with 
standards of participant opinion, which generally call for survey measurement at 
some point after participants have He sufficient experience with the program, 
Hypotheses dealing with growth require measurements from at least two points in 


time in order to assess change. This approach utilizes a pretest-posttest or repeated 


measures schedule. It is important to allow enaugh time between the test 
administrations so that the desired growth reasonably can be expectgd to occur. 

Hypotheses dealing with comparison groups require a simultaneous test 
administration after all groups have participated in their respective programs for the 
specified period of time. This is a ‘posttest-only schedule. If the groups are 
quasi-experimental, then all groups also must be pretested before the program begins 
so that initial differences can be taken into consideration. If the Sroups are true 
experimental, it still is desirable to pretest in order to enhance precision and 
minimize the weakening effects of dropouts during the program. 

Thug, instrument content must match program objectives.’ And, in designing 
the schedule of instrument administration, it is important to provide for the timely 
collection of data required to test the stated hypotheses. 
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DATA SYSTEMS g We , 

The creation of 4 data system capable of act onimiadtiny al eéllectat 
information is an important support task in the evaluation process. The absence of a 
systematic approach to -data storage and maintenance greatly increases the 
occurrence of lost, irretrievable or unusable information. As soon as the evaluation 
design has-been finalized, construction of data systems can be undertaken: The 


hypotheses, subject groups and instruments all serve to define the parameters of the — 


systerk which will meet the needs. 

. The first task-is to establish’an identification system for members of subject 
groups. ‘Tt usually’ is pfeferable to employ a’ numerical system to minimize the 
recognizability of individuals, excépt by designated persons who have the translation 
lists. Subject numbers can be constructed to include group identification, time of 
program ‘entry ‘or any other variable which might be helpful. in file categorization. 
Whatever the gumbering procedure selected, it is important to allow room for group 
, members who may be added in the future and to assure that each subject will have a 
unique number. 


The construction of a aboot framework establishes one dimension of the 


data system; it identifies the range of individuals across subject groups. The other 
major dimension is the specification of information to be collected within each 
subject group. Such data consist of the.results obtained from all of the instruments 
administered to each group and may be in the form of individual item scores, 
subscale scores, total scores or any combination. 


A basic information file might be diagrammed as follows in siuuke 1. The first 


- column lists the range of members in each group. The other columns list information 
and scores obtained for each individual. Most systems wil] be more complicated than 
this example because they will include more variables and multiple administrations 

-of instruments, but the diagram may serve as a model which can be expanded, _ 

The codes and formats for storing the data should be selected according to the 
information needs defined by the evaluation plan. The data system should be 
designed to facilitate the anticipated analyses by keeping the form and, location of 
all data clear and retrievable for evaluative use. “ 

After the data system has been outlined, the choice.of implementing it as a 
manual or automated system can be made. This decision depends upon both the size 
of the data files andthe complexity of planned analyses. Usually some degree of 
machine processing capability is desirable, which requires individual file formats 
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designed for use with a computer tard processing system or other automated pre 
cedures. 

Designing and implementing the data system is a task requiring technical 
expertise and experience With. the problems which typically are encountered. It - sys 
should be done with great care and informed advice. Like the other elements in the 
evaluation protess, the quality of thgedata system directly affects the clarity and 
usefylness of the eamation results. ‘ “< 
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ANALYSES a BS 


- The definition of hypotheses, subject groups and instruments is needed in tie 
design of an analysis plan. Analyses should be selected to describe the results clearly, 
to test hypotheses statistically with the ‘most rigor possible and to facilitate 
‘unambiguous i interpretation of the evaluation outcomes. Hypotheses determine what 

‘effects are to be tested. Subject groups constitute the experimental samples amiong 
whom effects are hypothesized. Instruments provide measures of the criteria 
selected to represent the hypothesized effects, and analyses are the statistical 
techniques which support o ueny the existence of effects within the hypothesis 

“framework. f 


Planning specific analyses depends greatly on the decisions made in previous 


. design stages, but some gerleral guidelines can be suggested. More specific 
information on analyses:i -is presented in the RBS Analysis Service Guide. 
The ‘first level ‘of aNalysis should, be descriptive. Appropriate distributional 
"statistics should be displayed for each subject group on all available measures. These 
data serve to depict group characteristics and Suggest between-group differences 
which may need to be considered. rf 
The next level of analysis is cull to uncover any differences between 
initially selected groups and the.groups available for final analysis. Since the groups 
’ were chosen to represent specific target populations, it is necessary to know how 
they changed in composition, over the course of the year. Initial groups will be 
* decreased in size both by attrition from the program and testing absence. It is 
important tb estimate the effects of such reductions in the samples by statistically 


comparing the subjects remaining for final analysis with those who have been ° 


eliminated. These comparisons should include any subject characteristics for which 
pretest information is available. The results will allow an: estimate’ of the 
representativeness of posttest data in terms of the initially drawn samples and may 
suggest subsidiary analyses in the hypothasis testing. Absence of such estimates 
constitutes a weakness in interpreting results whenever group attrition is substantial. 

When, an estimation of the representativeness of .available data has been 


provided through procedures such as those just outlined, the final level of analysis 


may be designed: the testing of hypotheses. Where a criterion or standard of success _ 


has been established for a subject\group, the group performance mean, or other 
répresentative statistic, may be compared directly with the designated standard. If 
development within groups has been hypothesized, then statistical tests comparing 
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the pretest and posttest performance levels m thay be ead ed. For hypothesized 
between-group “differences, analyses comparing the perfornrance of the various 
groups should be carried out. Selection of specific statistical techniques depends 
- upon the nature of the data and the questions posed. 

This general flow of analytic procedures provides descriptive information, 
assessment, of data representativeness and testing of the,stated hypotheses. The 
specific elements of the analysis plan should be designed well before the-unalyses are, 
conducted. This timing is important because the analysis ‘design serves’as input for 
implementing the’ necessary data systems, and also because - “unanticipated or 
unannouficed analyses may be viewed as seancring for desirable results, 
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REPORTING PROCEDURES | 


All of the steps in this evaluation, process contribute to the production of 

evaluation findings. These findings are communicated in reports which should be 

Geared to the audiences that’ will receive them. Three major audiences can be 
ntified: 1) participants in the program, 2) sponsors of the program or potential — 

_ sponsors of similar programs and 3) external education and research groups. For 

_ each group the pertinent questions and when they need to be answered must be 
specie? so that a schedule of reporting can be designed. 

- Participants in the program require the most detailed and frequent evaluation 
reporting. For example, staff will be able to use individual student results in guiding 
students through the program. Members of any: subject group will be interested in | 

_ overall results for their ‘group. Students will want to know: how they: ‘scored on 
‘achievement tests. Program leaders will want to-be alerted to apparent problems. 
Each of these possible reporting” ‘categories requires: a timely internal feedback 
system. Reporting in this sense isan ongoing communication activity, often without ~ 
much formal interpretation. It serves an important function in supporting the 
operation and development of the program, but it also necessitates a field test of the 
data collection, storing and manipulation procedures. Testing these procedures at an- 
early point can be helpful in avoiding problems later. 

Sponsors and potential sponsors usually require a different level of secaies 
tion and reporting. They are interested ih summary data on progress and outcomes 
as well as interpretations of the meaning of results. Typically this information calls 
for a mid-year and year-end report in which the evaluation process is described, ° 
results outlined clearly and concisely and outcomes interpreted in terms of program 

_ success and recommended future direction. Such reports also will be of interest to 
the program participants. 

Often it is valuable to prepare reports for external groups, re the 
program at a general level would:be useful at regional, state or national educational 
forums. Groups implementing similar programs may be given assistance through 
reports on problems encountered and solutions found..Reseasch and evaluation 
audiences might be interested in reports on tethnical issues and research significance. 
Such reporting must be designed to meet the needs of the particular audience. 

Reporting -is the final stage of the evaluation process. In many senses the 
report is the culmination of that process since all of the preceding stages combine to 
generate it. Reporting is the evaluation product. As such, it should be planned 

_ carefully to utilize available data to their fullest. 
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_ participants also will be administered an opinion survey. The skills tests will be: 
administered on a:pretest-posttest schedule; the opinion surveys will be given only - 
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RESOURCES REQUIRED FOR EVALUATION 


This mapual has presented an outline of the evaluation planning activities 
which are recommended for experimental or demonstrational programs. A final 
topic concerns the resources necessary for designing and conducting a worthwhile 


evaluation. Needed resources vary with the scope of the program objectives, 


_pninbers and sizes of subject groups and the complexity of analyses planned. For 


' : this reason projections must be fairly eneral, with substantial room for justment 
“e proj y 
to meet local needs. : ' 


Although the preceding sections of “this manual have dealt primarily wid 
evaluation planning tather than implementation issues and ‘problems, resource 
estimates for both planning and implementation are. included here. Implementation 
estimates are provided because such costs are generally 4 planning concern. 

In order to establish some basis for resource projections, a hypothetical: career, 
education program will be used. In this illustration it is assumed that approximately 


. 100 students, equally divided between experimental and control groups, are to be 


included in the evaluation. These- student samples would create: parent groups 
totaling at least 100 members. Since’ this program is to utilize community-based 
career education experiences, approximately 50 resource sites with a total of 100 
key site personnel would participate in the study. . 3 

The program objectives are assumed to focus on the development of career 
skills, life skills and basic academic skills. One major testing instrument is to be 
employed in each skill area along with a student background questionnaire. All 


once during the year.. 

Systematic feedback of evalhation results to program staff would be available, 
as would automated instrument scoring and a computer-based data system. Progress 
of the experimental group in each skill area over the course of the year will be 
analyzed, and the superiority or inferiority of the experimental group relative to the 
control group will be assessed. Opinions, perceptions and suggestidns of program 
participants are to be documented. Standard statistical procedures will be used for 
analysis purposes; all results will be presented in evaluation-reports. 

It is assumed that the services of a trained and experienced evaluator will be 
available locally to accomplish: most of the tasks. External evaluation technical 


assistance and services are projected to facilitate major steps in the evaluation ” 


process. . - 
ao 


wy, 


a : ; 
Given this hypothetical example, a generalized allocation of resources may be 
projected for evaluation planning and evaluation implementation. Figure 2 presents 


. projects for the planning process. . : 
. FIGURE 2 ‘ 
_ESTIMATED EVALUATION PLANNING RESOURCES. 
Task Task Area Staff Days Technical Assistance Days 
Sedo ‘ * : 
=F a Definition 1-2 
2 Evaluation ‘ Questions 3-4 \ -2 
~ 3. Statement of Hypotheses 1-2° a 7 
4. Subject Groups - 4-5 i=2 
_ 5. Instruments ; 2-3 1-2. ‘ 
6.. Data Systems . 5-6 1-2 
7. Analyses on 4-5 : 2.4 
8. Reporting Procedures 2-3 0-1 
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Estimates of time requirements ate included for each evaluation planning task. 
The “staff days” refer to the program evaluator, and “technical assistance days” 


. denote consulting services from: an external agency such as Research for Better 


Schools. Participation of other program staff has not been accounted for but will be 
necessary according to local needs. Support services and non-staff resources likewise 
have not been calculated because they are dependent on local gondhtiont and usually 
“can be readily extrapolated from the staff costs. 

ith these qualifications, it is estimated that the tliat planning resources 
needed Should approximate 22-30 staff days and 6-12.technical assistance days. 
These resodxce requirements are affected by the extensiveness of the program, but 


_they increase at a less than proportional rate. The planning and design tasks for a 


200-student program are not much different from the tasks for a 100-student 
program. In baa sense, phamne costs are much less variable than inplemieatiion 
costs. 
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Figure 3 presents projectiong for evaluation implementation that are based on 


the hypothetical program; changes in conditions. would have a directly Propetional 
effect gn sesources needed. ; 
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’ ESTIMATED EVALUATION. IMPLEMENT ATION RESOURCES 


Staff "Assistance . : 
- Task Areas , _ Days . _Days 4 Other Costs 
1. Subject Groups — implement, maintain ©. 3- 4 Oe ar 


bd . 


2. Instruments — purchase, administer, 


_scare . -" .10- . $1200 instruments and 


| . Pe 9 
3, Data Systems — implement, maintain $ 300 computer a 


$, 600 computer : 
services 


“ 


‘4. Analyses perform, interpret 


5. Reporting Procedures — prepare feed- 
back, interim, final and other reports 


2 


$ 600 production 
, 59- : $2700 


These accumulated implementation estimates suggest the need for an. 
_ approximately one-third-time staff evaluator supported by 9-14 outside technical 
assistance days and $2700 in other resources. Technical assistance, scoring, 
computer and production services are available from Research fot Better Schools. 
These resource estimates do not include support services, physical ne supplies 
and materials, postage and other non-staff costs. 

| It should be, emphasized that these resource ailoeericna: are pevsealted 
estimates. More precise projections would require planning information specific to 
the individual project to be evaluated. Expansion of the program objectives, student 
groups or intended analyses beyond the hypothetical example used. for these 
projections would necessitate proportionate increases in evaluation resources. 
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‘The création of an evaluation. , plan ‘requires a ‘crucial ae ebcihieally" 
demanding set of tasks in the evaluation process. The ‘scope and‘ Sophistication of the 
‘plan. do. much to determine.the usefulness - and conclusiveness of the” ‘evaluation 
‘findings. This manual. has attempted to discuss evaluation planning in a concise but’ 
comprehensive. way -by. structuring a series of fh major sequential steps. dnthis final | 
‘section, the steps Will erotic, and several overall concerns will'be noted: * 

The. plapping: process” begins . with defining: the _program to be evfluated) * 
ee - formulating evaluation questions and ‘refining the questions into testable: hypotheses. 
a » These’ steps establish the evalyation - weeds’ and fornially state expectations for the . 
“. “program. Next, the selection of subject groups and instruments enables the’testing of - 

‘ hypotheses by specifying the ¢ffeot’ variables, and the samples among f whom effects - 
~ eh are, intended, Finally, data o}siths must be designed to accommodate the evaluation 
‘gfinformation, and an ee plan must be developed to show how the hypotheses: 
(will be tested... ' 4 : : ; 
> * *. "+ Although .these ‘steps can be discussed “ ag separate — in the soaldaciait 
iS *'plarining "process, theif ‘interrelatedness “should not .be minimized. The decisions 

",made’ «at “each stage strorigly influence the’ requirements, of succeeding stages. 
* Likewise, difficulties in later stages may call for revisions at earlier points, Changes 
‘in the’ ptogram or evaluation components which affect one stage necessitate a review 
of the entire process to ensure consistency. The activities within evaluation planning | 
thus are interdependent ‘and must be conducted with that perspective. sh 
so i. The evaluation planning process is sufficiently complex to benefit from 
advisory assistance; an external review is always appropriate. Omissions, etrors of 
- judgment and inconsistencies -in evaluation’ planning generally are magnified and * 
‘harder to correct during evaluation implementation. Weaknesses in the design be- 
come limitations in the usefulness and interpretability of the findings. a 

The objectives which are to be accomplished by an evaluation effort vary. 
substantially from progrant to program. The intended role of evaluatiori may range . 
from simply completing a funding requirement to providing extensive. information in 
the operation and development of the’program. The evaluation planner may help to 
shape that role and must be aware of the real evaluation objectives as they relate to 
the program as a whole. It is important to have this awareness’ in the planning ; 
process in order to maximize the usefulness of evaluation results. 

Tlic evaluation planner also must be .able to deal, with non-evaluation factors 
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- which affect evaluation ‘design. i ; 
. impede evaluation plans, and in these cases flexibility and creative problem-solving ° 
“* skills are required to adapt to the environment while maintaining the integrity of the 


r 


- evaluation effort, 


pertinent results will be greatly enhanced. ~ 
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Other program activities or decisions may 
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The interpretation of results is 


insufficient. attention. A complete evaluation plan should project how major 


alternative outcomes will be interpreted should they be obtained.’This exercise in © 


projection accomplishes two important aim > It uncovers possible findings which 
would be -uninterpretable dnd may call for redesign.; It also establishes the potential 


: . Le . a 
implementation concern., =~: 


. After a satisfactory evaluation plan h been develo'ped, its implementation 


* can proceed. Implémentation introduces a whole series of problems and issues which 


could not be addressed in this manual. Even the best plans have limited value if they 
are not rigorously implemented. Successful implementation turns the potential of 
the design into reality, If evaluation planning and implementation are accorded 
proper’ attention, the probability-of obtaining -conclusive, unambiguous and 
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alter or * 


as planning issue which typically receives . 


significance: of the results. thie, interpretation ‘a planning as well as an | 
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